Systems and methods for predictive modelling of digital assessments with multi-model adaptive learning engine

ABSTRACT

Systems and methods are provided by which an adaptive learning engine may select a machine learning model service to determine a probability that a user will respond correctly to a given assessment item of a digital assessment on their first attempt. The adaptive learning engine may receive a request identifying the user, the assessment item, and request data. A model selector may generate a model reference corresponding to a model definition based on the request data. The feature data to be retrieved and/or calculated may be defined by the model definition. The feature data may be processed by a model service executing a machine learning model selected by the adaptive learning engine based on the model definition. Based on the probability output by the model, the adaptive learning engine may whether the user should be preemptively assigned credit for the assessment item.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from provisionalapplication No. 62/802,931, titled SYSTEMS AND METHODS FOR PREDICTIVEMODELLING OF DIGITAL ASSESSMENTS, filed Feb. 8, 2019, the entirecontents of which is incorporated herein by reference.

FIELD OF THE INVENTION

This disclosure relates to the field of systems and methods forevaluating the probability that a user will respond correctly to anassessment item of a digital assessment on their first attempt.Applications of this evaluation may include automatically giving creditfor assessment items with a high correct first attempt probability.

BACKGROUND OF THE INVENTION

A computer network or data network is a telecommunications network whichallows computers to exchange data. In computer networks, networkedcomputing devices exchange data with each other along network links(data connections). The connections between nodes are established usingeither cable media or wireless media.

Network computer devices that originate, route and terminate the dataare called network nodes. Nodes can include hosts such as personalcomputers, phones, servers as well as networking hardware. Two suchdevices can be said to be networked together when one device is able toexchange information with the other device, whether or not they have adirect connection to each other.

Computer networks differ in the transmission media used to carry theirsignals, the communications protocols to organize network traffic, thenetwork's size, topology and organizational intent. In most cases,communications protocols are layered on other more specific or moregeneral communications protocols, except for the physical layer thatdirectly deals with the transmission media.

Conventional electronic learning (i.e., e-learning) systems may providecontent and assessments to students, but may lack the ability to predictthe future performance of a student based on their previous interactionswith the systems.

SUMMARY OF THE INVENTION

The present invention provides systems and methods for the evaluation ofthe probability that a user will respond correctly to an assessment itemof a digital assessment on their first attempt.

In an example embodiment, a system may include a computer processor, anelectronic communication network, a feature data store, a modeldefinition data store, and a computer memory. The computer processor maybe in electronic communication with an external server via theelectronic communication network. The feature data store may be inelectronic communication with the computer processor. The modeldefinition data store may be in electronic communication with thecomputer processor and may include a plurality of machine learning modeldefinitions. A computer memory in electronic communication with thecomputer processor may be configured to store computer-readableinstructions which, when executed by the computer processor, cause thecomputer processor to receive, with a first service, a request from theexternal server, the request including a user identifier correspondingto a user, an assessment item identifier corresponding to an assessmentitem of a digital assessment, and request data, identifying, with amodel selector, a machine learning model based on the request data,retrieve, with a second service, a machine learning model definition ofthe plurality of machine learning model definitions from the modeldefinition data store, the machine learning model definitioncorresponding to the machine learning model, retrieve, with the secondservice, first feature data from the feature data store based on theuser identifier, the assessment item identifier, and the machinelearning model definition, and process, with a third service, the firstfeature data with the machine learning model to determine a correctfirst attempt probability for the user for the assessment item.

In some embodiments, the system may include an attempt data store inelectronic communication with the computer processor, and a useractivity analytics service configured to monitor interactions betweenuser devices and the system and to cause assessment data representingthe interactions to be stored in the attempt data store.

In some embodiments, the computer-readable instructions, when executedby the computer processor, may cause the computer processor to retrieve,with the second service, the assessment data from the attempt datastore, determine, with the second service, how to calculate secondfeature data from the assessment data based on the machine learningmodel definition, and calculate, with the second service, the secondfeature data from the assessment data. The third service may processboth the second feature data and the first feature data with the machinelearning model to determine the correct first attempt probability.

In some embodiments, the assessment data retrieved by the second servicemay represent one or more responses to assessment items submitted by theuser and one or more grades corresponding to the one or more responses.

In some embodiments, the system may include a global data storeconfigured to store global assessment data representing a plurality ofusers and at least one course, a course of the at least one course beingassociated with the assessment item. The system may include a globalfeature analytics service configured to retrieve the global assessmentdata from the global data store, generate the first feature data basedon the global assessment data, and cause the first feature data to bestored in the feature data store.

In some embodiments, the first feature data may include global correctfirst attempt rates calculated based on the global assessment data. Thesecond feature data may include user correct first attempt ratescalculated based on the assessment data.

In some embodiments, the plurality of users and the at least one courserepresented in the global assessment data are limited based on: usetype, user type, geographic region, course timing, or topic.

In an example embodiment, a method may include steps of receiving, witha first service executed by a processor of a first server, a requestfrom a second server, the request including a user identifiercorresponding to a user, an assessment item identifier corresponding toan assessment item of a digital assessment, and request data,referencing, with a model selector executed by the processor of thefirst server, the request data against a database to identify a machinelearning model associated with the request data in the database,retrieving, with a second service executed by the processor of the firstserver, a machine learning model definition from a model definition datastore, the machine learning model definition corresponding to themachine learning model, retrieving, with the second service, firstfeature data from a feature data store based on the user identifier, theassessment item identifier, and the machine learning model definition,and processing, with a third service executed by the first server, thefirst feature data with the machine learning model to determine acorrect first attempt probability for the user for the assessment item.

In some embodiments, the method may include steps of monitoring, with auser activity analytics service, interactions between a user deviceassociated with the user and a content distribution network thatincludes the first server, and storing, with the user activity analyticsservice, assessment data corresponding to the interactions in an attemptdata store in electronic communication with the user activity analyticsservice.

In some embodiments, the method may include steps of retrieving, withthe second service, the assessment data from the attempt data store,identifying, with the second service, instructions for calculatingsecond feature data, the instructions being included in the machinelearning model definition, calculating, with the second service, thesecond feature data based on the assessment data, and processing, withthe third service, the second feature data with the machine learningmodel in conjunction with processing the first feature data with themachine learning model to determine the correct first attemptprobability for the user for the assessment item.

In some embodiments, the assessment data may represent one or moreresponses to assessment items submitted by the user and one or moregrades corresponding to the one or more responses.

In some embodiments, the method may include steps of retrieving, with aglobal feature analytics service, global assessment data from a globaldata store, the global assessment data representing a plurality of usersand at least one course, a course of the at least one course beingassociated with the assessment item, generating, with the global featureanalytics service, the first feature data based on the global assessmentdata, and causing, with the global feature analytics service, the firstfeature data to be stored in the feature data store.

In some embodiments, the first feature data may include global correctfirst attempt rates calculated based on the global assessment data. Thesecond feature data may include user correct first attempt ratescalculated based on the assessment data.

In an example embodiment, a system may include at least one processorand at least one memory device. The at least one memory device may beconfigured to store computer-readable instructions which, when executed,cause the at least one processor to receive a request from an externalserver, analyze the request to identify a user and an assessment itemcorresponding to a digital assessment, retrieve a machine learning modeldefinition based on the request, retrieve first feature data based onthe user, the assessment item, and the machine learning modeldefinition, and execute the machine learning model to process at leastthe first feature data to determine a correct first attempt probabilityfor the user for the assessment item, the correct first attemptprobability representing a probability that a first response submittedby the user to the assessment item will be correct.

In some embodiments, the computer-readable instructions, when executed,may cause the at least one processor to retrieve assessment datarepresenting, in near-real-time, interactions between the user and acontent distribution network, and calculate second feature data based onthe assessment data according to instructions included in the machinelearning model definition.

In some embodiments, to execute the machine learning model, thecomputer-readable instructions, when executed, may further cause the atleast one processor to execute the machine learning model to process thefirst feature data and the second feature data to determine the correctfirst attempt probability for the user for the assessment item.

In some embodiments, the assessment data may represent one or moreresponses to assessment items submitted by the user and one or moregrades corresponding to the one or more responses.

In some embodiments, the computer-readable instructions, when executed,may cause the at least one processor to retrieve global assessment datarepresenting a plurality of users and at least one course, a course ofthe at least one course being associated with the assessment item,generate the first feature data based on the global assessment data, andcause the first feature data to be stored in a feature data store of theat least one memory device.

In some embodiments, the first feature data may include global correctfirst attempt rates for the plurality of users calculated based on theglobal assessment data. The second feature data may include user correctfirst attempt rates for the user calculated based on the assessmentdata.

In some embodiments, the system may include a model definition datastore that includes a plurality of machine learning model definitionsthat includes the machine learning model definition. Thecomputer-readable instructions, when executed, may cause the at leastone processor to identify a course and a title included in the request,perform a comparison of the course and the title to a database thatstores associations between a plurality of sets of courses and titlesand the plurality of machine learning model definitions that includesthe machine learning model definition, and determine that the machinelearning model of the plurality of machine learning models is associatedin the database with the course and the title. The machine learningmodel definition may correspond to the machine learning model.

The above features and advantages of the present invention will bebetter understood from the following detailed description taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system level block diagram showing one or more datastores, data centers, servers, and clients of a distributed computingenvironment, in accordance with an embodiment.

FIG. 2 illustrates a system level block diagram showing physical andlogical components of a special-purpose computer device within adistributed computing environment, in accordance with an embodiment.

FIG. 3 illustrates a block diagram of an example content distributionnetwork, in accordance with an embodiment.

FIG. 4 illustrates a block diagram of an example logical architecturefor the application of and acquisition of feature data for a selectedmachine learning model, in accordance with an embodiment.

FIG. 5 illustrates a process flow diagram for a method of identifyingassessment items that should be automatically marked correct based on auser's correct first attempt probability for those assessment items, inaccordance with an embodiment.

FIG. 6 illustrates a process flow diagram for a method of applying amachine learning model to determine a user's correct first attemptprobability for assessment items, in accordance with an embodiment.

FIG. 7 illustrates a process flow diagram for a method of determiningand storing global correct first attempt rates based on a machinelearning model definition, in accordance with an embodiment.

FIG. 8 illustrates a process flow diagram for a method of determiningand storing user correct first attempt rates based on a machine learningmodel definition, in accordance with an embodiment.

FIG. 9 illustrates a process flow diagram for a method of identifyingassessment items that should be automatically marked correct innear-real-time, in accordance with an embodiment.

DETAILED DESCRIPTION

The present invention will now be discussed in detail with regard to theattached drawing figures that were briefly described above. In thefollowing description, numerous specific details are set forthillustrating the Applicant's best mode for practicing the invention andenabling one of ordinary skill in the art to make and use the invention.It will be obvious, however, to one skilled in the art that the presentinvention may be practiced without many of these specific details. Inother instances, well-known machines, structures, and method steps havenot been described in particular detail in order to avoid unnecessarilyobscuring the present invention. Unless otherwise indicated, like partsand method steps are referred to with like reference numerals.

FIG. 1 illustrates a non-limiting example of a distributed computingenvironment 100, which includes one or more computer server computingdevices 102, one or more client computing devices 106, and othercomponents that may implement certain embodiments and features describedherein. Other devices, such as specialized sensor devices, etc., mayinteract with client 106 and/or server 102. The server 102, client 106,or any other devices may be configured to implement a client-servermodel or any other distributed computing architecture.

Server 102, client 106, and any other disclosed devices may becommunicatively coupled via one or more communication networks 120.Communication network 120 may be any type of network known in the artsupporting data communications. As non-limiting examples, network 120may be a local area network (LAN; e.g., Ethernet, Token-Ring, etc.), awide-area network (e.g., the Internet), an infrared or wireless network,a public switched telephone networks (PSTNs), a virtual network, etc.Network 120 may use any available protocols, such as (e.g., transmissioncontrol protocol/Internet protocol (TCP/IP), systems networkarchitecture (SNA), Internet packet exchange (IPX), Secure Sockets Layer(SSL), Transport Layer Security (TLS), Hypertext Transfer Protocol(HTTP), Secure Hypertext Transfer Protocol (HTTPS), Institute ofElectrical and Electronics (IEEE) 802.11 protocol suite or otherwireless protocols, and the like.

The embodiments shown in FIGS. 1-2 are thus one example of a distributedcomputing system, which is not intended to be limiting. The subsystemsand components within the server 102 and client devices 106 may beimplemented in hardware, firmware, software, or combinations thereof.Various different subsystems and/or components 104 may be implemented onserver 102. Users operating the client devices 106 may initiate one ormore client applications to use services provided by these subsystemsand components. Various different system configurations are possible indifferent distributed computing systems 100 and content distributionnetworks. Server 102 may be configured to run one or more serversoftware applications or services, for example, web-based or cloud-basedservices, to support content distribution and interaction with clientdevices 106. Users operating client devices 106 may in turn utilize oneor more client applications (e.g., virtual client applications) tointeract with server 102 to utilize the services provided by thesecomponents. Client devices 106 may be configured to receive and executeclient applications over one or more networks 120. Such clientapplications may be web browser based applications and/or standalonesoftware applications, such as mobile device applications. Clientdevices 106 may receive client applications from server 102 or fromother application providers (e.g., public or private applicationstores).

As shown in FIG. 1, various security and integration components 108 maybe used to manage communications over network 120 (e.g., a file-basedintegration scheme or a service-based integration scheme). Security andintegration components 108 may implement various security features fordata transmission and storage, such as authenticating users orrestricting access to unknown or unauthorized users,

As non-limiting examples, these security components 108 may comprisededicated hardware, specialized networking components, and/or software(e.g., web servers, authentication servers, firewalls, routers,gateways, load balancers, etc.) within one or more data centers in oneor more physical location and/or operated by one or more entities,and/or may be operated within a cloud infrastructure.

In various implementations, security and integration components 108 maytransmit data between the various devices in the content distributionnetwork 100. Security and integration components 108 also may use securedata transmission protocols and/or encryption (e.g., File TransferProtocol (FTP), Secure File Transfer Protocol (SFTP), and/or Pretty GoodPrivacy (PGP) encryption) for data transfers, etc.).

In some embodiments, the security and integration components 108 mayimplement one or more web services (e.g., cross-domain and/orcross-platform web services) within the content distribution network100, and may be developed for enterprise use in accordance with variousweb service standards (e.g., the Web Service Interoperability (WS-I)guidelines). For example, some web services may provide secureconnections, authentication, and/or confidentiality throughout thenetwork using technologies such as SSL, TLS, HTTP, HTTPS, WS-Securitystandard (providing secure SOAP messages using XML encryption), etc. Inother examples, the security and integration components 108 may includespecialized hardware, network appliances, and the like (e.g.,hardware-accelerated SSL and HTTPS), possibly installed and configuredbetween servers 102 and other network components, for providing secureweb services, thereby allowing any external devices to communicatedirectly with the specialized hardware, network appliances, etc.

Computing environment 100 also may include one or more data stores 110,possibly including and/or residing on one or more back-end servers 112,operating in one or more data centers in one or more physical locations,and communicating with one or more other devices within one or morenetworks 120. In some cases, one or more data stores 110 may reside on anon-transitory storage medium within the server 102. In certainembodiments, data stores 110 and back-end servers 112 may reside in astorage-area network (SAN). Access to the data stores may be limited ordenied based on the processes, user credentials, and/or devicesattempting to interact with the data store.

With reference now to FIG. 2, a block diagram of an illustrativecomputer system is shown. The system 200 may correspond to any of thecomputing devices or servers of the network 100, or any other computingdevices described herein. In this example, computer system 200 includesprocessing units 204 that communicate with a number of peripheralsubsystems via a bus subsystem 202. These peripheral subsystems include,for example, a storage subsystem 210, an I/O subsystem 226, and acommunications subsystem 232.

One or more processing units 204 may be implemented as one or moreintegrated circuits (e.g., a conventional micro-processor ormicrocontroller), and controls the operation of computer system 200.These processors may include single core and/or multicore (e.g., quadcore, hexa-core, octo-core, ten-core, etc.) processors and processorcaches. These processors 204 may execute a variety of resident softwareprocesses embodied in program code, and may maintain multipleconcurrently executing programs or processes. Processor(s) 204 may alsoinclude one or more specialized processors, (e.g., digital signalprocessors (DSPs), outboard, graphics application-specific, generalpurpose graphics processing units (GPGPUs), and/or other processors).

Bus subsystem 202 provides a mechanism for intended communicationbetween the various components and subsystems of computer system 200.Although bus subsystem 202 is shown schematically as a single bus,alternative embodiments of the bus subsystem may utilize multiple buses.Bus subsystem 202 may include a memory bus, memory controller,peripheral bus, and/or local bus using any of a variety of busarchitectures (e.g. Industry Standard Architecture (ISA), Micro ChannelArchitecture (MCA), Enhanced ISA (EISA), Video Electronics StandardsAssociation (VESA), and/or Peripheral Component Interconnect (PCI) bus,possibly implemented as a Mezzanine bus manufactured to the IEEE P1386.1standard).

I/O subsystem 226 may include device controllers 228 for one or moreuser interface input devices and/or user interface output devices,possibly integrated with the computer system 200 (e.g., integratedaudio/video systems, and/or touchscreen displays), or may be separateperipheral devices which are attachable/detachable from the computersystem 200. Input may include keyboard or mouse input, audio input(e.g., spoken commands), motion sensing, gesture recognition (e.g., eyegestures), etc.

As non-limiting examples, input devices may include a keyboard, pointingdevices (e.g., mouse, trackball, and associated input), touchpads, touchscreens, scroll wheels, click wheels, dials, buttons, switches, keypad,audio input devices, voice command recognition systems, microphones,three dimensional (3D) mice, joysticks, pointing sticks, gamepads,graphic tablets, speakers, digital cameras, digital camcorders, portablemedia players, webcams, image scanners, fingerprint scanners, barcodereaders, 3D scanners, 3D printers, laser rangefinders, eye gaze trackingdevices, medical imaging input devices, MIDI keyboards, digital musicalinstruments, and the like.

In general, use of the term “output device” is intended to include allpossible types of devices and mechanisms for outputting information fromcomputer system 200 to a user or other computer. For example, outputdevices may include one or more display subsystems and/or displaydevices that visually convey text, graphics and audio/video information(e.g., cathode ray tube (CRT) displays, flat-panel devices, liquidcrystal display (LCD) or plasma display devices, projection devices,touch screens, etc.), and/or non-visual displays such as audio outputdevices, etc. As non-limiting examples, output devices may include,indicator lights, monitors, printers, speakers, headphones, automotivenavigation systems, plotters, voice output devices, modems, etc.

Computer system 200 may comprise one or more storage subsystems 210,comprising hardware and software components used for storing data andprogram instructions, such as system memory 218 and computer-readablestorage media 216.

System memory 218 and/or computer-readable storage media 216 may storeprogram instructions that are loadable and executable on processor(s)204. For example, system memory 218 may load and execute an operatingsystem 224, program data 222, server applications, client applications220, Internet browsers, mid-tier applications, etc.

System memory 218 may further store data generated during execution ofthese instructions. System memory 218 may be stored in volatile memory(e.g., random access memory (RAM) 212, including static random accessmemory (SRAM) or dynamic random access memory (DRAM)). RAM 212 maycontain data and/or program modules that are immediately accessible toand/or operated and executed by processing units 204.

System memory 218 may also be stored in non-volatile storage drives 214(e.g., read-only memory (ROM), flash memory, etc.) For example, a basicinput/output system (BIOS), containing the basic routines that help totransfer information between elements within computer system 200 (e.g.,during start-up) may typically be stored in the non-volatile storagedrives 214.

Storage subsystem 210 also may include one or more tangiblecomputer-readable storage media 216 for storing the basic programmingand data constructs that provide the functionality of some embodiments.For example, storage subsystem 210 may include software, programs, codemodules, instructions, etc., that may be executed by a processor 204, inorder to provide the functionality described herein. Data generated fromthe executed software, programs, code, modules, or instructions may bestored within a data storage repository within storage subsystem 210.

Storage subsystem 210 may also include a computer-readable storage mediareader connected to computer-readable storage media 216.Computer-readable storage media 216 may contain program code, orportions of program code. Together and, optionally, in combination withsystem memory 218, computer-readable storage media 216 maycomprehensively represent remote, local, fixed, and/or removable storagedevices plus storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation.

Computer-readable storage media 216 may include any appropriate mediaknown or used in the art, including storage media and communicationmedia, such as but not limited to, volatile and non-volatile, removableand non-removable media implemented in any method or technology forstorage and/or transmission of information. This can include tangiblecomputer-readable storage media such as RAM, ROM, electronicallyerasable programmable ROM (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disk (DVD), or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or other tangible computer readablemedia. This can also include nontangible computer-readable media, suchas data signals, data transmissions, or any other medium which can beused to transmit the desired information and which can be accessed bycomputer system 200.

By way of example, computer-readable storage media 216 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 216 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 216 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magneto-resistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 200.

Communications subsystem 232 may provide a communication interface fromcomputer system 200 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 2, thecommunications subsystem 232 may include, for example, one or morenetwork interface controllers (NICs) 234, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 236, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally and/or alternatively, the communicationssubsystem 232 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, Fire Wire® interfaces, USB® interfaces, and the like.Communications subsystem 236 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

In some embodiments, communications subsystem 232 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 200. For example,communications subsystem 232 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators). Additionally, communications subsystem 232 maybe configured to receive data in the form of continuous data streams,which may include event streams of real-time events and/or event updates(e.g., sensor data applications, financial tickers, network performancemeasuring tools, clickstream analysis tools, automobile trafficmonitoring, etc.). Communications subsystem 232 may output suchstructured and/or unstructured data feeds, event streams, event updates,and the like to one or more data stores that may be in communicationwith one or more streaming data source computers coupled to computersystem 200.

The various physical components of the communications subsystem 232 maybe detachable components coupled to the computer system 200 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 200. Communicationssubsystem 232 also may be implemented in whole or in part by software.

Due to the ever-changing nature of computers and networks, thedescription of computer system 200 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

With reference now to FIG. 3, a block diagram is shown illustratingvarious components of a content distribution network (CDN) 300 whichimplements and supports certain embodiments and features describedherein. In some embodiments, the content distribution network 300 mayinclude one or several physical components and/or one or several virtualcomponents such as, for example, one or several cloud computingcomponents. In some embodiments, the content distribution network 300may include a mixture of physical and cloud computing components.

Content distribution network 300 may include one or more contentmanagement servers 302. Content management servers 302 may include anydesired type of server including, for example, a rack server, a towerserver, a miniature server, a blade server, a mini rack server, a mobileserver, an ultra-dense server, a super server, or the like, and mayinclude various hardware components, for example, a motherboard, aprocessing units, memory systems, hard drives, network interfaces, powersupplies, etc. For example, the content management servers 302 maycorrespond to the computer server 102 of FIG. 1. Content managementserver 302 may include one or more server farms, clusters, or any otherappropriate arrangement and/or combination or computer servers. Contentmanagement server 302 may act according to stored instructions locatedin a storage subsystem (e.g., storage subsystem 210 of FIG. 2) of theserver 302, and may run an operating system, including any commerciallyavailable server operating system and/or any other operating systemsdiscussed herein.

The content distribution network 300 may include one or more data storeservers 304, such as database servers and file-based storage systems.The data store servers 304 can access data that can be stored on avariety of hardware components.

Data store servers 304 may comprise stored data relevant to thefunctions of the content distribution network 300. In some embodiments,multiple data stores may reside on a single server 304, either using thesame storage components of server 304 or using different physicalstorage components to assure data security and integrity between datastores. In other embodiments, each data store may have a separatededicated data store server 304.

Content distribution network 300 also may include one or more userdevices 306 and/or supervisor devices 310. User devices 306 andsupervisor devices 310 may display content received via the contentdistribution network 300, and may support various types of userinteractions with the content. User devices 306 and supervisor devices310 may include mobile devices such as smartphones, tablet computers,personal digital assistants, and wearable computing devices. Such mobiledevices may run a variety of mobile operating systems, and may beenabled for Internet, e-mail, short message service (SMS), Bluetooth®,mobile radio-frequency identification (M-RFID), and/or othercommunication protocols. Other user devices 306 and supervisor devices310 may be general purpose personal computers or special-purposecomputing devices including, by way of example, personal computers,laptop computers, workstation computers, projection devices, andinteractive room display systems. Additionally, user devices 306 andsupervisor devices 310 may be any other electronic devices, such as athin-client computers, an Internet-enabled gaming systems, business orhome appliances, and/or a personal messaging devices, capable ofcommunicating over network(s) 320.

In different contexts of content distribution networks 300, user devices306 and supervisor devices 310 may correspond to different types ofspecialized devices, for example, student devices and teacher devices inan educational network, employee devices and presentation devices in acompany network, different gaming devices in a gaming network,clinician/teacher devices and patient/student devices in a clinicaldiagnosis or learning classification network, etc. Additionally,different user devices 306 and supervisor devices 310 may be assigneddifferent designated roles, such as presenter devices, teacher devices,clinician devices, administrator devices, or the like, and in such casesthe different devices may be provided with additional hardware and/orsoftware components to provide content and support user capabilities notavailable to the other devices.

The content distribution network 300 also may include a privacy server308 that maintains private user information at the privacy server 308while using applications or services hosted on other servers. Forexample, the privacy server 308 may be used to maintain private data ofa user within one jurisdiction even though the user is accessing anapplication hosted on a server (e.g., the content management server 302)located outside the jurisdiction. In such cases, the privacy server 308may intercept communications between a user device 306 or supervisordevice 310 and other devices that include private user information. Theprivacy server 308 may create a token or identifier that does notdisclose the private information and may use the token or identifierwhen communicating with the other servers and systems, instead of usingthe user's private information.

The content distribution network 300 may include one or morecommunication networks 320. Although only a single network 320 isidentified in FIG. 3, the content distribution network 300 may includeany number of different communication networks between any of thecomputer servers and devices shown in FIG. 3 and/or other devicesdescribed herein. Communication networks 320 may enable communicationbetween the various computing devices, servers, and other components ofthe content distribution network 300. Various implementations of contentdistribution networks 300 may employ different types of networks 320,for example, computer networks, telecommunications networks, wirelessnetworks, and/or any combination of these and/or other networks.

As illustrated in FIG. 3, the content management server 302 may be incommunication with one or more additional servers, such as a contentserver 312, an administrator server 314, and/or a user data server 316.Each of these servers may include some or all of the same physical andlogical components as the content management server(s) 302, and in somecases, the hardware and software components of these servers 312-316 maybe incorporated into the content management server(s) 302, rather thanbeing implemented as separate computer servers.

Content server 312 may include hardware and software components togenerate, store, and maintain the content resources for distribution touser devices 306 and other devices in the network 300. For example, incontent distribution networks 300 used for professional training andeducational purposes, or clinical diagnosis of students/patents, thecontent server 312 may include data stores of training materials,presentations, plans, syllabi, reviews, evaluations, interactiveprograms and simulations, course models, course outlines, assessmentsand diagnostic modules, and various training interfaces that correspondto different materials and/or different types of user devices 306.

Administrator server 314 may include hardware and software components toinitiate various administrative functions at the content managementserver 302 and other components within the content distribution network300. For example, the administrator server 314 may monitor device statusand performance for the various servers, data stores, and/or userdevices 306 in the content distribution network 300. When necessary, theadministrator server 314 may add or remove devices from the network 300,and perform device maintenance such as providing software updates to thedevices in the network 300. Various administrative tools on theadministrator server 314 may allow authorized users to set user accesspermissions to various content resources, monitor resource usage byusers and devices 306, and perform analyses and generate reports onspecific network users and/or devices (e.g., resource usage trackingreports, training evaluations, etc.).

User data server 316 may include hardware and software components thatstore and process data for multiple users relating to each user'sactivities and usage of the content distribution network 300. Forexample, the content management server 302 may record and track eachuser's system usage, including their user device 306, content resourcesaccessed, and interactions with other user devices 306. This data may bestored and processed by the user data server 316, to support usertracking and analysis features. For instance, in the contexts ofprofessional training, education, and/or clinical diagnosis of studentsor patients, the user data server 316 may store and analyze digitalassessments completed by each user or training materials viewed,presentations attended, courses or tests completed, the user's responses(i.e., submitted answers) to assessment items (i.e., questions orprompts) of the digital assessments (e.g., homework assignments, tests,formative assessments, summative assessments, and/or the like) or otherinteractions, assessment or evaluation results, and/or the like.Individual digital assessments may be included as part of a title, whichmay correspond to a particular course or textbook for a particularsubject. A title may include multiple digital assessments. The digitalassessments may be organized into objectives, which may be organizedinto sections, which may be organized into chapters. Each digitalassessment may include a number of assessment items to which users mayinteract with and submit responses to. For example, each title mayinclude one or more chapters, each chapter may include one or moresections, each sections may include one or more objectives, eachobjective may include one or more digital assessments, and each digitalassessment may include one or more assessment items.

For example, when a user (sometimes referred to herein in this contextas a responder) wholly or partially completes a digital assessment, theresponses to each of the assessment items of the digital assessment thatare responded to by the user may be stored in the user data server 316(e.g., as response data). It should be understood that the methodsdescribed herein by which response data and corresponding user andassessment identifier information are stored are intended to beillustrative and not limiting. If desired, alternative organizationaldata storage paradigms may be used. As used herein, an “assessment item”refers to the smallest measurable part of any activity with built-inassessment (e.g., a tutorial, a formative, summative, or diagnostic testor quiz, or any other applicable activity). Assessment items mayinclude, for example, selected response items (e.g., multiple-choice ortrue-or-false questions), and/or constructed response items (e.g., fillin the blank questions or graphing questions). The user data server 316may then analyze the responses and generate grades for the responsescorresponding to whether each response is correct or incorrect. In someembodiments, correct first attempt grades may be generated, which may beindicative of whether a user's first attempt at responding to anassessment item is correct (e.g., in which case a grade of “1” isgenerated) or incorrect (e.g., in which case a grade of “0” isgenerated). The generated grades may be stored in the data server 316.In some embodiments, assessment items may provide a user with the optionof accessing a hint before responding to the assessment item. For eachresponse stored in the data server 316, a corresponding aid usage valuemay also be stored (e.g., in a corresponding data entry), which may beused to indicate whether the user accessed a defined type of learningaid (e.g., a hint, an example problem and corresponding solution, and/ora guided solution) before responding to the corresponding assessmentitem. For example, an aid usage value of 1 indicates that aid of thedefined type was utilized by the user when responding to thecorresponding assessment item, while an aid usage value of 0 indicatesthat that type of aid was not utilized.

In some embodiments, the data store servers 304 may store assessmentdata corresponding to one or more digital assessments of one or moretitles (e.g., books or courses). Assessment data stored in the datastore servers 304 may include responses, grades, aid usage values, andresponder (i.e., user) information (e.g., including user identifiers)for the assessment items the corresponding digital assessments and/ortitles. For example, assessment data corresponding to an individualassessment item (which may sometimes be referred to as the assessmentitem data for that assessment item) may include a dataset (e.g., alook-up table (LUT)) of responders that have responded to the assessmentitem. Each entry of the dataset may correspond to a different responderand may include the user ID of the responder, an assessment itemidentifier from which the assessment item, assessment, objective,section, chapter, and title of the corresponding assessment item may bedetermined, a grade for the response (e.g., 1 for a correct answer and 0for an incorrect answer), and an aid usage value for the response (e.g.,1 if aid of a defined type was used and 0 otherwise).

Assessment data corresponding to a given user and/or assessment datacorresponding to one or more pieces of content across one or morehierarchical content levels (e.g., title, chapter, section, objective,assessment, and assessment item levels) may be analyzed (e.g., by one ormore computer processors such as processors 204 of FIG. 2) in order todetermine the probability (sometimes referred to herein as a correctfirst attempt probability) that the given user will respond correctly toa given assessment item on their first attempt. In some embodiments, a“first attempt” or “first attempt response” may be defined as the firstresponse submitted by a user to an assessment item, and would notinclude subsequent responses submitted by the user to the assessmentitem. In other embodiments, a “first attempt” may be defined as thefirst predefined number of responses a user submits to an assessmentitem (e.g., the first three responses), where if any of the firstpredefined number of responses are correct, the user's “first attempt”is considered to be correct. Machine learning models, which may includea random forest model, may be trained and applied to perform thisdetermination of correct first attempt probability, given a set offeature data corresponding to a user and an assessment item for whichthe correct first attempt probability is being determined, as will bedescribed.

Correct first attempt probability, as a metric, may have many practicalapplications. For example, a user that has been assigned a set ofassessment items to complete (e.g., as a homework assignment) mayalready be skilled enough to correctly answer a portion of the assignedassessment items. Rather than have the user respond to assessment itemsthat the user likely already knows how to correctly answer, a system mayprocess assessment data related to the user and the assessment items(e.g., using a trained machine learning model) to determine a respectivecorrect first attempt probability value for the user for each of theassessment items. The system may then identify assessment itemscorresponding to a correct first attempt probability value that exceedsa predetermined threshold (e.g., 70%, 80%, or 90%). These identifiedassessment items may then automatically be given credit (e.g., marked ascorrect) without requiring the user to submit a response. In this way, auser may avoid having to answer assessment items that they are alreadylikely capable of answering correctly, and the user may spend more timeanswering assessment items that may be more difficult for them, so thatthe user may focus more on content that they need to practice, the usermay save time, and homework may be made more efficient. In someembodiments, a given instructor may access an instructor user interface(e.g., of a web browser or application) via an instructor device (e.g.,personal computer, mobile/smart phone, etc.), and through thisinstructor interface may adjust this predetermined threshold, or anyother threshold described herein. Additionally, the given instructor mayselect whether or not to enable the automatic assignment of credit forstudent assignments in this way by modifying (e.g., toggling) acorresponding setting via interaction with the instructor userinterface.

FIG. 4 shows a system 400 that may be implemented by one or more servers(e.g., servers 112 and data stores 110 of FIG. 1; system 200 of FIG. 2,servers 302, 304, 312, 314, and/or 316 of FIG. 3) to collect assessmentdata, generate feature data from the feature data according to one ormore model definitions, retrieve feature data corresponding to a modeldefinition of a selected machine learning model, processing theretrieved feature data using the selected machine learning model toproduce a result (e.g., a correct first attempt probability), andgenerating a prediction based on the result (e.g., a prediction ofwhether a given user will respond correctly to an assessment item). Forexample, the system 400 may be included in the content distributionnetwork 300 of FIG. 3, and may be communicatively coupled to orpartially implemented by any applicable devices thereof.

The system 400 includes an adaptive learning engine (ALE) 403, machinelearning model services 406 (sometimes referred to as “model services406”), data stores 411, a user activity analytics service 418, a userfeature analytics service 420, a global feature analytics service 422,and a global data store 424. The ALE 403 includes an ALE service 404, aprediction evaluator 405, a model selector 407, and a feature retrievalservice 408. The data stores 411 include a feature store 412 (sometimesreferred to as “feature data store 412”), a model definition store 414(sometimes referred to as “model definition data store 414”), and anattempt store 416 (sometimes referred to as “attempt data store 416”).Some or all of the ALE service 404, the feature retrieval service 408,and each of the machine learning model services 406 may includerepresentational state transfer (REST) application programminginterfaces (APIs). The data stores 411 may be maintained on one or morememory devices (e.g., storage subsystem 210 of FIG. 2) of one or more ofthe servers that implement the system 400.

In some embodiments, some or all of the ALE engine 403, the data stores411, the user activity analytics service 418, the user feature analyticsservice 420, the global feature analytics service 422, and the globaldata store 424 may be implemented by one or more processors and/ormemory devices associated with one or several servers (e.g., servers 112and data stores 110 of FIG. 1; system 200 of FIG. 2, servers 302, 304,312, 314, and/or 316 of FIG. 3). For example, the ALE engine 403 may beexecuted by a first processor running on a first server (e.g., one ofthe content management servers 302, FIG. 3), the data stores 411 and/orthe global store 424 may be stored in one or more memory devices of thefirst server or may be included in one or more data store servers (e.g.,data store servers 304, FIG. 3) in electronic communication with thefirst server, The user activity analytics service 418 may be implementedby one or more processors of the first server, or by those of anadministrative server (e.g., administrative server 314, FIG. 3) coupledto the first server. The user feature analytics service 420 may beexecuted by one or more processors of the first server, or by those of auser data server (e.g., user data server 316, FIG. 3) coupled to thefirst server. The global feature analytics service 422 may be executedby one or more processors of the first server, or by those of a userdata server (e.g., user data server 316, FIG. 3) coupled to the firstserver. Each of the model services 406 may be executed by one or moreprocessors of the first server, or by those of a user data server (e.g.,user data server 316, FIG. 3) coupled to the first server. In someembodiments, the first server may instead be a server cluster, serverfarm, or a cloud-based server. In some embodiments, the external server402 may be a second server that is in electronic communication to thefirst server (e.g., a second server of the content management servers302, FIG. 3).

An external server 402 may be in electronic communication with the ALEservice. For example, the external server 402 may be coupled to one ormore servers (e.g., system 200 of FIG. 2) executing the ALE service 404through an electronic communication network (e.g., networks 120, 320,FIGS. 1 and 3). The external server 402 may send a request to the ALEservice 404. The request may, for example, identify a user, a set ofassessment items, and request data that may be used as a basis forselecting a machine learning model to be used in calculating correctfirst try probabilities for the user for the assessment items. Forexample, the request data may identify a title and course that therequest corresponds to. In some embodiments, the request data may alsoidentify a custom user group that includes the user associated with therequest.

The model selector 407 may receive the request data from the ALE service404. The model selector 407 may reference the request data against themodel definition store 414 to identify a model reference 410corresponding to one of the model definitions stored in the modeldefinition store 414. For example, the model selector 407 may comparethe title, course, and/or custom user group included in the request datato a database or look-up table (LUT) included in the model definitionstore 414, which stores predetermined associations between such requestdata (e.g., combinations of title, course, and/or custom user group) andvarious model definitions. Upon identifying a model definition that isassociated in the database or LUT with the request data received fromthe external server 402 via the ALE service 404, the model selector maysend a model reference 410 to the feature retrieval service 408. Forexample, the model reference 410 may identify which model definitionshould be retrieved by the feature retrieval service 408.

A model definition may be loaded (e.g., manually loaded) into the modeldefinition store 414 in order to populate the model definition store414. The feature retrieval service 408 may later retrieve a modeldefinition from the model definition store 414 based on the modelreference 410 received from the model selector 407.

In some embodiments, a default machine learning model may beautomatically selected for use in processing the request, and the modelreference 410 retrieved by the model selector 407 from the modeldefinition store 414 may correspond to the default machine learningmodel. The corresponding model definition that is retrieved by thefeature retrieval service 408 based on the model reference 410 maydefine which feature data should be provided as inputs to the selectedmachine learning model (e.g., to a corresponding model service of themodel services 406 that is configured to execute the selected machinelearning model).

Machine learning models available for selection may include models basedon different algorithms, such as random forest, artificial neuralnetworks, kernel principal component analysis, decision trees withboosting, logistic regression, or any other applicable machine learningmodel algorithm. Additionally, machine learning models available forselection may include models that use the same algorithm, but that havebeen trained using different sets of data. For example, differentmachine learning models could be trained using only assessment datalimited to one or more of: content corresponding to a particular usetype (e.g., quizzes, study plans, tests, and/or homework), userscorresponding to a particular user type (e.g., underperforming(“struggling”) learners, moderately skilled learners, and/or expertlearners), content corresponding to a particular course or title, userscorresponding to a particular geographic region, user/content pairscorresponding to a particular course timing (e.g., whether a givencourse occurs during spring, summer, or fall, though other type of timestratifications, such as the particular months or years in which coursesare offered, may be used), or content corresponding to a particulartopic (e.g., mathematics, history, language, etc.).

Once a model definition has been retrieved from the model data store414, the feature retrieval service 408 may retrieve correspondingfeature data from the feature data store 412 of the data stores 411according to the model definition. The retrieved feature data may beprovided to a selected machine learning model service of the machinelearning model services 406, which inputs the retrieved feature data tothe selected machine learning model for processing. The selected machinelearning model service may be selected by the ALE service 404 based onthe machine learning model definition retrieved by the feature retrievalservice 408. For example, the feature retrieval service may provide themachine learning model definition that it has retrieved from the modeldefinition store 414 to the ALE service 404, which the ALE service 404may use as a basis for selecting one of the machine learning modelservices 406 that is configured to execute the machine learning modelthat corresponds to that machine learning model definition. Uponexecution, the machine learning model, which may be executed via theselected model service of the model services 406, may output arespective correct first attempt probability for the user for each ofthe assessment items defined in the request received from the externalserver 402. The correct first attempt probabilities may be provided tothe prediction evaluator 405, which may compare each correct firstattempt probability to a predefined threshold (e.g., 70%, 80%, or 90%).If the prediction evaluator 405 determines that a given correct firstattempt probability exceeds the predefined threshold, the ALE service404 may send a recommendation to the external server 402, indicatingthat credit should automatically be given to the user for the assessmentitem corresponding to the given correct first attempt probability (e.g.,that assessment item should automatically be marked correct). In someembodiments, the external server 402 may then automatically give theuser credit for assessment items according to recommendations receivedfrom the ALE service 404. Alternatively, an administrator (or some otherprivileged user) may manually identify which of the assessment itemsrecommended by the ALE service 404 actually receive automatic credit.

Regarding the generation of the feature data stored in the feature datastore 412, the feature data may be organized into one or more databases,and may be retrieved and input to selected machine learning modelsexecuted at a selected machine learning model service of the machinelearning model service 406. In some embodiments, the feature data mayinclude correct first attempt rates. These correct first attempt ratesmay be determined for an individual user (in which case they arereferred to herein as user correct first attempt rates) or acrossmultiple (e.g., all) responders (in which case they are referred toherein as global correct first attempt rates) for one or morehierarchical levels of content.

A given user correct first attempt rate may be calculated (e.g.,periodically according to a schedule) by a user feature analyticsservice 420 based on grades of “first attempt” responses that weresubmitted to assessment items of a particular piece of content. Forexample, the user correct first attempt rate for given content may becalculated as an average of the grades of “first attempt” responsessubmitted to the assessment items of the content by the individual user.For example, for user correct first attempt rates, this content mayinclude titles that the user has wholly or partially completed, chaptersthat the user has wholly or partially completed, the last fiveassessment items completed by the user, the last three assessment itemscompleted by the user, and the last assessment item completed by theuser.

As used in the present example, user “wholly” completes a piece ofcontent when they have submitted responses to all assessment items ofthe content, and “partially” completes the content when they havesubmitted responses to only a portion of the assessment items of thecontent. Content referred to simply as having been “completed” by theuser may be assumed to refer to “wholly” completed content.

A given global correct first attempt rate may be calculated (e.g.,periodically according to a schedule) by a global feature analyticsservice 422 based on the grades of all “first attempt” responsessubmitted by multiple (e.g., all) responders to assessment items of thecorresponding content. For example, for global correct first attemptrates, this content may be defined as a title, chapter, section,objective, assessment, or individual assessment item. In someembodiments, a respective global correct first attempt rate may becalculated (e.g., by the global feature analytics service 422) for eachpiece of content at each hierarchical level for which assessment data isavailable. For example, the global correct first attempt rate for agiven piece of content may be calculated as an average of the grades ofthe “first attempt” responses submitted to the assessment items of thecontent by all responders that have submitted responses to the content.

In some embodiments, a given response submitted by an individual userthat would otherwise be considered a “first attempt” response may beomitted from the determination of the user or global correct firstattempt rates if the aid usage value of that response indicates that aidof the determined type was utilized by the user prior to submitting theresponse (i.e., the aid usage value equals 1). In this way, initialresponses submitted after an “excessive” amount of aid has been receivedby a corresponding user for the corresponding assessment items may beomitted from these calculations, as such aid may cause a given initialresponse to not accurately reflect the ability of the correspondinguser.

Correct first attempt rates may be periodically determined on a per-userand/or global basis (i.e., calculated as described above by theanalytics services 420 and 422 and subsequently stored in the featurestore server 412. The global feature analytics service 422 may retrieveglobal assessment data (e.g., assessment data corresponding to allusers, not just for an individual user) from a global data store 424,which may store assessment data for all users and all content of thesystem. The user feature analytics service 420 may retrieve userassessment data from one or more attempt data stores 416. The attemptdata store 416 may be an individual attempt data store that storesassessment data for one or more individual users. This assessment datamay include any responses that a given user has submitted to content andcorresponding grades for those responses. The assessment data mayfurther identify (e.g., via one or more content identifiers) one or morelevels of content to which each response corresponds, which may includethe assessment item to which the response corresponds and/or thechapter, section, title, objective, assessment, and/or homeworkassignment that include(s) that assessment item. The assessment data mayfurther include, for a given response, a user identifier that identifiesthe user who submitted the response. In some embodiments, the assessmentdata may include time data indicating when a given response wassubmitted by the user. In this way, the most recent responses toassessment items submitted by the user may be identified by the system.The attempt data store 416 may be updated by a user activity analyticsservice 418, which may monitor and record the user's interactions withcontent of the system. In some embodiments, the attempt data store 416may be updated by the user activity analytics service 418 in real time.In contrast, the global data store 424 may be updated in scheduled batchjobs. As an example, each entry in the global data store 424 may definea user, a response submitted by the user, a grade for the content (e.g.,correct or incorrect), the assessment item corresponding to theresponse, one or many content identifiers of hierarchical content levelsassociated with the assessment item (e.g., the title/course, section,chapter, objective, homework assignment, and/or assessment that includesthe assessment item), whether the user responded to the assessment itemcorrectly on their first try, and/or whether a corresponding learningaid was opened by the user before submitting the response. In someembodiments, the global data store 424 and/or the global featureanalytics service 422 may be configured such that only datacorresponding to selected books, courses, titles, or other content types(e.g., which may be defined in a configuration file associated with theglobal data store 424) are retrieved by and included in the global datastore 424, or otherwise made available for use in global featurecalculation by the global feature analytics service 422.

The particular correct first attempt rates that are calculated by theanalytics services 420 and 422 (or by the feature retrieval service 408,in some embodiments) may be determined based on all possible correctfirst attempt rates that could be required by model definitions of themodel definition store 414.

A given model definition of the model definition store 414 maycorrespond to a machine learning model that is configured (e.g.,trained) to generate a prediction regarding a given user with respect totheir expected performance when responding to a given piece of content(e.g., likelihood of responding correctly to an assessment item on theuser's first attempt). For example, a model definition may require asfeature data: user correct first attempt rates for the title level andthe chapter level, for the last (i.e., most recent) five assessmentitems completed by the user, for the last three assessment itemscompleted by the user, and for the last assessment item completed by theuser, global correct first attempt rates for the title level, thechapter level, the section level, the objective level, the assessmentlevel, and/or the assessment-item level, an average amount of time takenby the user to complete the last assessment item completed by the user,an average amount of time taken by the user to complete assessment itemsat the title level, the chapter level, over the last five assessmentitems completed by the user, and/or over the last three assessment itemscompleted by the user, an item response theory (IRT) ability/skill levelof the user calculated for one or more corresponding chapters, sections,and/or titles, an average number of attempts until a correct response issubmitted by the user for the title level, the chapter level, thesection level, the objective level, the assessment level, and/or theassessment item level, and/or correct first attempt rates of the user oncontent that is defined as prerequisite for the content for which aprediction is being made.

Based on one or more of the model definitions stored in the modeldefinition store 414, the user feature analytics service 420 mayperiodically determine user correct first attempt rates for: all titlesfor each individual user (e.g., such that a user correct first attemptrate is calculated for every possible user-title pair for whichassessment data exists), for all chapters for each individual user(e.g., such that a user correct first attempt rate is calculated forevery possible user-chapter pair for which assessment data exists, forthe five assessment items that each individual user has most recentlyresponded to, for the three assessment items that each individual userhas most recently responded to, and for the assessment item that eachindividual user has most recently responded to. Alternatively, asdiscussed below, the feature retrieval service may determine these usercorrect first attempt rates in near-real-time based on user activitydata stored in the attempt store 416 (e.g., which itself may be updatedto include user activity data or assessment data in near-real time).

Based on one or more of the model definitions stored in the modeldefinition store 414, the global feature analytics service 422 mayperiodically determine, based on responses submitted by all applicableusers, global correct first attempt rates for: all titles, all chapters,all sections, and all assessment items. These correct first attemptrates may be stored in the feature data store 412 as they arecalculated. For example, the services 420 and 422 may be performed togenerate feature data periodically according to a predetermined schedule(e.g., once every hour to once every twelve hours).

In an illustrative alternate embodiment, feature calculation may beperformed in real time, rather than being performed in batches accordingto a predetermined schedule. In such embodiments, the user featureanalytics service 420 may be omitted, and the feature retrieval service408 may be executed instead to generate features based on user-specificdata based on staged data (sometimes referred to herein as “stagedassessment data”) stored in the attempt store 416. The feature retrievalservice 408 may determine which features to calculate and how tocalculate those features based on the model definition of the modeldefinition store 414 corresponding to the model reference 410, which mayinclude instructions for how user-specific features are to becalculated. The global feature analytics service 422 may still calculateglobal feature data in batches based on data stored in the global datastore 424, and may still store global features in the feature data store412. However, user-specific features calculated by the feature retrievalservice 408 that are calculated in near-real-time (e.g., with a delay ofaround 30 to 60 seconds, rather than a delay of around 6 hours) may bepassed directly to the selected machine learning model service of themachine learning model services 406 as they are calculated, rather thanbeing stored in the feature data store 412. The attempt store 416,instead of storing detailed data describing each response submitted byeach user, may store aggregated or “staged” data for individual useractivity at one or more content levels. For example, rather than storinga separate entry for each questioned answered by a given user in a givenchapter (e.g., or other hierarchical content level, such as section,title, objective, or assessment), the attempt store 416 may includestaged data which may include a total number of attempted answerssubmitted by the given user for questions in the given chapter and atotal number of correct answers submitted by the given user for thequestions in the given chapter. In some embodiments, other data may beincluded in the staged data, such as the number of hints used within agiven hierarchical content level. Such staged data may be updated innear-real-time, as users submit responses to questions, which enablesthe feature retrieval service 408 to calculate corresponding features innear-real-time. By calculating user-specific features in near-real-timein this way, predictions made by the selected machine learning modelservice of the machine learning model services 406 may be moreresponsive and representative of recent user behavior. Additionally, byusing staged data instead of including separate data entries for eachresponse submitted by a user, less data has to be stored by the system,and processing time (e.g., for feature calculation) may be reduced.

Turning now to FIG. 5, an illustrative method 500 is shown by whichassessment items that should be automatically given credit may beidentified in response to a request received from an external server(e.g., server 402 of FIG. 4). The method 500 may be performed byexecuting instructions stored in a non-transitory computer-readablememory device (e.g., of storage subsystem 210 of FIG. 2) with one ormore computer processors (e.g., processors 204 of FIG. 2). For example,the method 500 may be performed by the system 400 of FIG. 4.

At step 502, an ALE service (e.g., ALE service 404 of FIG. 4) receives arequest from an external server (e.g., server 402 of FIG. 4). Therequest may identify a user (e.g., via a user identifier), one or moreassessment items (e.g., via one or more assessment item identifiers),and request data (e.g., which may identify a test, course, and/or customuser group) to be processed.

At step 504, a machine learning model service (e.g., of machine learningmodel services 406 of FIG. 4) executes a machine learning model, whichmay be trained prior to the execution of the method 500, to determine acorrect first attempt probability for the identified user for each ofthe identified assessment items. The machine learning model andcorresponding model service may be identified by the ALE service basedon a machine learning model definition that is retrieved by a featureretrieval service (e.g., feature retrieval service 408, FIG. 4) based ona model reference (e.g., model reference 410, FIG. 4) that is generatedby a model selector (e.g., model selector 407, FIG. 4) based on therequest data (e.g., by referencing the request data against a databaseor LUT of the model definition data store that stores associationsbetween titles, courses, and/or custom user groups and machine learningmodel definitions). For example, the machine learning model service mayprocess feature data retrieved from a feature data store (e.g., featuredata store 412 of FIG. 4), the feature data corresponding to one or moreuser correct first response rates and/or global correct first responserates. For example, the feature data may be retrieved by the featureretrieval service, then passed to the machine learning model service viathe ALE service. The particular feature data input to the machinelearning model service may be determined based on the identified user,the identified assessment item for which the correct first attemptprobability is being determined, and the machine learning modeldefinition corresponding to the model reference.

At step 506, a prediction evaluator (e.g., prediction evaluator 405 ofFIG. 4) compares the correct first response probabilities to apredefined threshold (e.g., 70%, 80%, or 90%), and the ALE service mayidentify any assessment items having correct first responseprobabilities that exceed the predefined threshold. In some embodiments,for example, the predefined threshold may be a value that is set by theinstructor of a corresponding course, or a system administrator.

At step 508, the ALE service sends a recommendation to the externalserver, indicating that the identified assessment items having correctfirst response probabilities above the predefined threshold should beautomatically marked as correct (e.g., assigned credit) withoutrequiring a response from the user. In some embodiments, the recommendedautomatic assignment of credit and/or the sending of the recommendationmay only be carried in response to the ALE service determining that acorresponding setting (e.g., an adaptive homework setting or adaptiveassessment setting) is enabled in a configuration file stored in amemory device that is in electronic communication with the processor orprocessors executing the method 500. The configuration file may bemodified by an instructor or administrator to enable or disable theautomatic assignment of credit for one or more assessments, courses, orother hierarchical levels of content.

Turning now to FIG. 6, an illustrative method 600 is shown by which acorrect first attempt probability may be determined for a selected userand assessment item via the application of a selected, trained machinelearning model. The method 600 may be performed by executinginstructions stored in a non-transitory computer-readable memory device(e.g., of storage subsystem 210 of FIG. 2) with one or more computerprocessors (e.g., processors 204 of FIG. 2). For example, the method 600may be performed by the system 400 of FIG. 4. For example, the method600 may be performed in connection with the performance of steps 502 and504 of the method 500 of FIG. 5.

At step 602, a feature retrieval service (e.g., feature retrievalservice 408 of FIG. 4) receives a user identifier, an assessment itemidentifier, and request data from an ALE service (e.g., ALE service 404of FIG. 4) based on a request from an external server (e.g., server 402of FIG. 4).

At step 604, the feature retrieval service retrieves a machine learningmodel definition based on a model reference (e.g., model reference 410of FIG. 4), which may be generated by a model selector based on therequest data, from a machine learning model definition data store (e.g.,model definition data store 414 of FIG. 4). For example, the modelselector may first reference a database or LUT of the model definitiondata store using the request data to determine which machine learningmodel definition is associated with the request data. The request datamay identify a title, course, and/or custom user group, which may becollectively associated, in the database against which the request datais referenced, with a machine learning model definition included in themachine learning model definition data store. Then the model selectormay generate a model reference that identifies the machine learningmodel definition of the machine learning model definition data storethat is associated with the request data. The feature retrieval servicemay then retrieve the identified machine learning model definition fromthe machine learning model data store.

At step 606, the feature retrieval service retrieves feature data from afeature data store (e.g., feature data store 412 of FIG. 4) based on theuser identifier, the assessment item identifier, and the retrievedmachine learning model definition. The feature data of the feature datastore may be periodically updated according to the method 700 of FIG. 7and/or the method 800 of FIG. 8, for example. In some embodiments, theuser-specific portion of the feature data may be calculated in near-realtime according to at least a portion of the method 900 of FIG. 9, forexample. The feature data may include user correct first attempt ratesand global correct first attempt rates.

At step 608, a machine learning model service (e.g., a machine learningmodel service of the machine learning model services 406 of FIG. 4)processes the retrieved feature data with the machine learning model todetermine a correct first attempt probability for the assessment item,which corresponds to an expected likelihood that the user will respondcorrectly to the assessment item on their first attempt. The machinelearning model service may be selected from a group of machine learningmodel services by the ALE service based on the machine learning modeldefinition that was retrieved by the feature retrieval service.

It should be understood that the method 600 may be repeated for eachassessment item included in a request from the external server, if therequest identifies multiple assessment items.

Turning now to FIG. 7, an illustrative method 700 is shown by whichglobal correct first attempt rates may be calculated and stored asfeature data in a feature data store according to a machine learningmodel definition. The method 700 may be performed by executinginstructions stored in a non-transitory computer-readable memory device(e.g., of storage subsystem 210, FIG. 2) with one or more computerprocessors (e.g., processors 204, FIG. 2). For example, the method 700may be performed by the system 400 of FIG. 4.

At step 702, a feature retrieval service (e.g., feature retrievalservice 408, FIG. 4) retrieves a machine learning model definitioncorresponding to a model reference (e.g., model reference 410, FIG. 4)from a machine learning model definition data store (e.g., modeldefinition data store 414, FIG. 4). The model reference may be generatedby a model selector (e.g., model selector 407, FIG. 4) based on requestdata included in a request received by an ALE service (e.g., ALE service404, FIG. 4).

At step 704, an analytics service (e.g., global feature analyticsservice 422 of FIG. 4) retrieves assessment data from a global datastore (e.g., global data store 424 of FIG. 4).

At step 706, the global feature analytics service determines, based onthe assessment data, respective global correct first attempt rates foreach piece of content corresponding to each piece of content representedin the assessment data that corresponds to hierarchical content levelsrequired by the machine learning model definition. For example, themachine learning model definition may require global correct firstattempt rates for content at the title or course level, the chapterlevel, the section level, the objective level, the assessment level, andthe assessment-item level. In some embodiments, the content required bythe machine learning model definition may only include contentcorresponding to the title/course, chapter, section, objective, and/orassessment that include the assessment item or assessment items forwhich a correct first attempt probability or correct first attemptprobabilities are to be calculated by the machine learning model. Theglobal feature analytics service would then determine separate globalcorrect first attempt rates for each title of the assessment data, eachchapter of the assessment data, each section of the assessment data, andeach assessment item of the assessment data. As described previously,the global correct first attempt rate for a given piece of content maybe calculated as an average of the grades of all “first attempt”responses submitted to assessment items of the given piece of content.

At step 708, the global correct first attempt rates is provided to andstored at a feature data store (e.g., feature data store 412 of FIG. 4)as feature data.

It should be understood that method 700 may be performed for eachmachine learning model definition of the machine learning model datastore, and may be performed periodically according to a schedule.

Turning now to FIG. 8, an illustrative method 800 is shown by which usercorrect first attempt rates may be calculated and stored as feature datain a feature data store according to a machine learning modeldefinition. The method 800 may be performed by executing instructionsstored in a non-transitory computer-readable memory device (e.g., ofstorage subsystem 210, FIG. 2) with one or more computer processors(e.g., processors 204, FIG. 2). For example, the method 800 may beperformed by the system 400 of FIG. 4.

At step 802, a feature retrieval service (e.g., feature retrievalservice 408, FIG. 4) retrieves a machine learning model definitioncorresponding to a model reference (e.g., model reference 410, FIG. 4)from a machine learning model definition data store (e.g., modeldefinition data store 414 of FIG. 4). The model reference may begenerated by a model selector (e.g., model selector 407, FIG. 4) basedon request data included in a request received by an ALE service (e.g.,ALE service 404, FIG. 4).

At step 804, an analytics service (e.g., user feature analytics service420 of FIG. 4) retrieves assessment data from an attempt data store(e.g., attempt data store 416 of FIG. 4). The attempt data store mayinclude assessment data corresponding to individual users and theirresponses to assessment items. This assessment data may be updated(e.g., in real time or near-real-time) by a user activity analyticsservice (e.g., user activity analytics service 418 of FIG. 4) thattracks a user's activity and records responses submitted by the user,along with other related assessment data such as timestamps for responsesubmissions and the grades of responses submitted by the user.

At step 806, the feature analytics service determines, for the user,user correct first attempt rates for each piece of content required bythe machine learning model definition. For example, for a given user,the machine learning model definition may require, as feature data, usercorrect first attempt rates for the title level and the chapter level,for the last (i.e., most recent) five assessment items completed by theuser, for the last three assessment items completed by the user, and forthe last assessment item completed by the user, an average amount oftime taken by the user to complete the last assessment item completed bythe user, an average amount of time taken by the user to completeassessment items at the title level, the chapter level, over the lastfive assessment items completed by the user, and/or over the last threeassessment items completed by the user, an item response theory (IRT)ability/skill level of the user calculated for one or more correspondingchapters, sections, and/or titles, an average number of attempts until acorrect response is submitted by the user for the title level, thechapter level, the section level, the objective level, the assessmentlevel, and/or the assessment item level, and/or correct first attemptrates of the user on content that is defined as prerequisite for thecontent for which a prediction is being made. As described previously,the user correct first attempt rate for a given piece of content may becalculated as an average of the grades of all “first attempt” responsessubmitted to assessment items of the given piece of content. Thisexample is meant to be illustrative and not limiting, and it should beunderstood that more or fewer types of feature data may be required byvarious machine learning model definitions.

At step 808, the user correct first attempt rates is provided to andstored at a feature data store (e.g., feature data store 412 of FIG. 4)by the feature analytics service as at least part of a collection offeature data (sometimes referred to as a “feature vector”) associatedwith the model definition.

It should be understood that method 800 may be performed for eachmachine learning model definition of the machine learning model datastore, and may be performed periodically according to a schedule.

Turning now to FIG. 9, an illustrative method 900 is shown by which asystem may determine whether to automatically provide credit to a userfor selected assessment items based on user correct first attempt ratesthat are calculated in near-real-time according to a machine learningmodel definition. The method 900 may be performed by executinginstructions stored in a non-transitory computer-readable memory device(e.g., of storage subsystem 210 of FIG. 2) with one or more computerprocessors (e.g., processors 204 of FIG. 2). The method 900 of FIG. 9may, for example, correspond to the alternate embodiment described inconnection with FIG. 4, in which the user feature analytics service 420is omitted, and user-specific feature data is calculated innear-real-time by the feature retrieval service 408 based on stagedassessment data retrieved from the attempt store 416. For example, themethod 900 may be initiated in response to an ALE service (e.g., ALEservice 404, FIG. 4) receiving a corresponding request from an externalserver (e.g., external server 402, FIG. 4) that is coupled to the serverthat executes the ALE service (e.g., as in step 502 of FIG. 5).

At step 902, a feature retrieval service (e.g., feature retrievalservice 408, FIG. 4) retrieves a machine learning model definitioncorresponding to a model reference (e.g., model reference 410, FIG. 4)from a machine learning model definition data store (e.g., modeldefinition data store 414, FIG. 4). The model reference may be generatedby a model selector (e.g., model selector 407, FIG. 4) based on requestdata included in the request received by the ALE service. The machinelearning model definition may include definitions of which feature datais required for processing by the corresponding machine learning model,definitions of how at least a portion of that feature data (e.g.,user-specific feature data) is to be calculated.

At step 904, the feature retrieval service retrieves staged assessmentdata from an attempt data store (e.g., attempt data store 416, FIG. 4).The attempt data store may include staged assessment data correspondingto individual users and their responses to assessment items. Thisassessment data may be updated (e.g., in real time or near-real-time) bya user activity analytics service (e.g., user activity analytics service418 of FIG. 4) that tracks a user's activity and records responsessubmitted by the user, along with other related assessment data such asthe grades of responses submitted by the user. The feature retrievalservice may determine which staged assessment data to retrieve based oninstructions included in the machine learning model definition.

At step 906, the feature retrieval service determines, for the user,user correct first attempt rates for each piece of content, as requiredby the machine learning model definition. For example, for a given user,the machine learning model definition may require user correct firstattempt rates for the title level and the chapter level, for the last(i.e., most recent) five assessment items completed by the user, for thelast three assessment items completed by the user, and for the lastassessment item completed by the user, an average amount of time takenby the user to complete the last assessment item completed by the user,an average amount of time taken by the user to complete assessment itemsat the title level, the chapter level, over the last five assessmentitems completed by the user, and/or over the last three assessment itemscompleted by the user, an item response theory (IRT) ability/skill levelof the user calculated for one or more corresponding chapters, sections,and/or titles, an average number of attempts until a correct response issubmitted by the user for the title level, the chapter level, thesection level, the objective level, the assessment level, and/or theassessment item level, and/or correct first attempt rates of the user oncontent that is defined as prerequisite for the content for which aprediction is being made. As described previously, the user correctfirst attempt rate for a given piece of content may be calculated as anaverage of the grades of all “first attempt” responses submitted toassessment items of the given piece of content. This example is meant tobe illustrative and not limiting, and it should be understood that moreor fewer types of feature data may be required by various machinelearning model definitions

At step 908, the user correct first attempt rates is provided by thefeature retrieval service to a machine learning model service (e.g., ofmodel services 406, FIG. 4) as part of a feature vector to be input toand processed by a machine learning model corresponding to the machinelearning model definition, that is executed by the machine learningmodel service. The machine learning model service may be selected basedon the machine learning model definition.

At step 910, the machine learning model service generates one or morecorrect first attempt probabilities based on the feature data of thefeature vector that was provided by the feature retrieval service, whichincludes the correct first attempt rates. For example, the machinelearning model service may generate a different correct first attemptprobability for each assessment item being considered (e.g., withrespect to whether credit should be automatically assigned for the userfor those assessment items).

At step 912, a prediction evaluator (e.g., prediction evaluator 405,FIG. 4) receives the correct first attempt probability or probabilitiesfrom the machine learning model service, then compares the correct firstattempt probability or probabilities to a corresponding threshold (e.g.,70%, 80%, 90%, which may be defined as a default setting by the system,or which may be adjusted/adjustable by an instructor or administrator).Based on the results of the comparison(s), the prediction evaluatoridentifies any correct first attempt probabilities that exceed thethreshold.

At step 914, the ALE service (e.g., ALE service 404, FIG. 4) receivesthe results of the comparison(s) and/or the identified correct firstattempt probabilities found to exceed the threshold. The ALE servicethen identifies the assessment items associated with each of theidentified correct first attempt probabilities found to exceed thethreshold.

At step 916, the ALE service sends a recommendation to an externalserver (e.g., external server 402, FIG. 4; which may be the sameexternal server that sent the request that initiated the method 900).The recommendation may indicate that the identified assessment itemsshould be automatically marked as correct (e.g., assigned credit)without requiring a response from the user. In some embodiments, therecommended automatic assignment of credit and/or the sending of therecommendation may only be carried in response to the ALE servicedetermining that a corresponding setting (e.g., an adaptive homeworksetting or adaptive assessment setting) is enabled in a configurationfile stored in a memory device that is in electronic communication withthe processor or processors executing the method 900. The configurationfile may be modified by an instructor or administrator to enable ordisable the automatic assignment of credit for one or more assessments,courses, or other hierarchical levels of content.

It should be understood that method 900 may be performed innear-real-time, such that feature data generated via the method 900 mayreflect user activity that has occurred within a recent time frame(e.g., within around 30 to 60 seconds of receipt of the request). Inthis way, the method 900 may provide recommendations for automaticcredit assignment for a user that reflects recent activity of the userin addition to historical activity of the user and/or global activity ofother users.

It should be understood that the automatic assignment of credit forassessment items and/or homework assignments discussed herein may becarried out in a variety of ways, but may generally be performed byupdating, with one or more computer processors, a database (e.g., whichmay be stored in the user data server 316 and/or data store servers 304of FIG. 3) in which user grades are stored to add or modify an entry ofthe database to record that a given user has received credit for a givenassessment item or homework assignment.

Other embodiments and uses of the above inventions will be apparent tothose having ordinary skill in the art upon consideration of thespecification and practice of the invention disclosed herein. Thespecification and examples given should be considered exemplary only,and it is contemplated that the appended claims will cover any othersuch embodiments or modifications as fall within the true scope of theinvention.

The Abstract accompanying this specification is provided to enable theUnited States Patent and Trademark Office and the public generally todetermine quickly from a cursory inspection the nature and gist of thetechnical disclosure and in no way intended for defining, determining,or limiting the present invention or any of its embodiments.

The invention claimed is:
 1. A system comprising: a computer processor;an electronic communication network, the computer processor being inelectronic communication with an external server via the electroniccommunication network; a feature data store in electronic communicationwith the computer processor; a model definition data store that is inelectronic communication with the computer processor and that includes aplurality of machine learning model definitions; and a computer memoryin electronic communication with the computer processor and configuredto store computer-readable instructions which, when executed by thecomputer processor, cause the computer processor to: receive, with afirst service, a request from the external server, the request includinga user identifier corresponding to a user, an assessment item identifiercorresponding to an assessment item of a digital assessment, and requestdata; identifying, with a model selector, a machine learning model basedon the request data; retrieve, with a second service, a machine learningmodel definition of the plurality of machine learning model definitionsfrom the model definition data store, the machine learning modeldefinition corresponding to the machine learning model; retrieve, withthe second service, first feature data from the feature data store basedon the user identifier, the assessment item identifier, and the machinelearning model definition; and process, with a third service, the firstfeature data with the machine learning model to determine a correctfirst attempt probability for the user for the assessment item.
 2. Thesystem of claim 1, further comprising: an attempt data store inelectronic communication with the computer processor; and a useractivity analytics service configured to monitor interactions betweenuser devices and the system and to cause assessment data representingthe interactions to be stored in the attempt data store.
 3. The systemof claim 2, wherein the computer-readable instructions, when executed bythe computer processor, cause the computer processor to: retrieve, withthe second service, the assessment data from the attempt data store;determine, with the second service, how to calculate second feature datafrom the assessment data based on the machine learning model definition;and calculate, with the second service, the second feature data from theassessment data, wherein the third service processes both the secondfeature data and the first feature data with the machine learning modelto determine the correct first attempt probability.
 4. The system ofclaim 3, wherein the assessment data retrieved by the second servicerepresents one or more responses to assessment items submitted by theuser and one or more grades corresponding to the one or more responses.5. The system of claim 4, further comprising: a global data storeconfigured to store global assessment data representing a plurality ofusers and at least one course, wherein a course of the at least onecourse is associated with the assessment item; and a global featureanalytics service configured to: retrieve the global assessment datafrom the global data store; generate the first feature data based on theglobal assessment data; and cause the first feature data to be stored inthe feature data store.
 6. The system of claim 5, wherein the firstfeature data includes global correct first attempt rates calculatedbased on the global assessment data, and wherein the second feature dataincludes user correct first attempt rates calculated based on theassessment data.
 7. The system of claim 6, wherein the plurality ofusers and the at least one course represented in the global assessmentdata are limited based on: use type, user type, geographic region,course timing, or topic.
 8. A method comprising: receiving, with a firstservice executed by a processor of a first server, a request from asecond server, the request including a user identifier corresponding toa user, an assessment item identifier corresponding to an assessmentitem of a digital assessment, and request data; referencing, with amodel selector executed by the processor of the first server, therequest data against a database to identify a machine learning modelassociated with the request data in the database; retrieving, with asecond service executed by the processor of the first server, a machinelearning model definition from a model definition data store, themachine learning model definition corresponding to the machine learningmodel; retrieving, with the second service, first feature data from afeature data store based on the user identifier, the assessment itemidentifier, and the machine learning model definition; and processing,with a third service executed by the first server, the first featuredata with the machine learning model to determine a correct firstattempt probability for the user for the assessment item.
 9. The methodof claim 8, further comprising: monitoring, with a user activityanalytics service, interactions between a user device associated withthe user and a content distribution network that includes the firstserver; and storing, with the user activity analytics service,assessment data corresponding to the interactions in an attempt datastore in electronic communication with the user activity analyticsservice.
 10. The method of claim 9, further comprising: retrieving, withthe second service, the assessment data from the attempt data store;identifying, with the second service, instructions for calculatingsecond feature data, the instructions being included in the machinelearning model definition; calculating, with the second service, thesecond feature data based on the assessment data; and processing, withthe third service, the second feature data with the machine learningmodel in conjunction with processing the first feature data with themachine learning model to determine the correct first attemptprobability for the user for the assessment item.
 11. The method ofclaim 10, wherein the assessment data represents one or more responsesto assessment items submitted by the user and one or more gradescorresponding to the one or more responses.
 12. The method of claim 11,further comprising: retrieving, with a global feature analytics service,global assessment data from a global data store, the global assessmentdata representing a plurality of users and at least one course, whereina course of the at least one course is associated with the assessmentitem; generating, with the global feature analytics service, the firstfeature data based on the global assessment data; and causing, with theglobal feature analytics service, the first feature data to be stored inthe feature data store.
 13. The method of claim 12, wherein the firstfeature data includes global correct first attempt rates calculatedbased on the global assessment data, and wherein the second feature dataincludes user correct first attempt rates calculated based on theassessment data.
 14. A system comprising: at least one processor; and atleast one memory device configured to store computer-readableinstructions which, when executed, cause the at least one processor to:receive a request from an external server; analyze the request toidentify a user and an assessment item corresponding to a digitalassessment; retrieve a machine learning model definition based on therequest; retrieve first feature data based on the user, the assessmentitem, and the machine learning model definition; and execute the machinelearning model to process at least the first feature data to determine acorrect first attempt probability for the user for the assessment item,the correct first attempt probability representing a probability that afirst response submitted by the user to the assessment item will becorrect.
 15. The system of claim 14, wherein the computer-readableinstructions, when executed, cause the at least one processor to:retrieve assessment data representing, in near-real-time, interactionsbetween the user and a content distribution network; and calculatesecond feature data based on the assessment data according toinstructions included in the machine learning model definition.
 16. Thesystem of claim 15, wherein, to execute the machine learning model, thecomputer-readable instructions, when executed, further cause the atleast one processor to: execute the machine learning model to processthe first feature data and the second feature data to determine thecorrect first attempt probability for the user for the assessment item.17. The system of claim 16, wherein the assessment data represents oneor more responses to assessment items submitted by the user and one ormore grades corresponding to the one or more responses.
 18. The systemof claim 16, wherein the computer-readable instructions, when executed,cause the at least one processor to: retrieve global assessment datarepresenting a plurality of users and at least one course, wherein acourse of the at least one course is associated with the assessmentitem; generate the first feature data based on the global assessmentdata; and cause the first feature data to be stored in a feature datastore of the at least one memory device.
 19. The system of claim 18,wherein the first feature data includes global correct first attemptrates for the plurality of users calculated based on the globalassessment data, and wherein the second feature data includes usercorrect first attempt rates for the user calculated based on theassessment data.
 20. The system of claim 14, further comprising: a modeldefinition data store comprising a plurality of machine learning modeldefinitions that includes the machine learning model definition, whereinthe computer-readable instructions, when executed, cause the at leastone processor to: identify a course and a title included in the request;perform a comparison of the course and the title to a database thatstores associations between a plurality of sets of courses and titlesand the plurality of machine learning model definitions that includesthe machine learning model definition; and determine that the machinelearning model of the plurality of machine learning models is associatedin the database with the course and the title, wherein the machinelearning model definition corresponds to the machine learning model.